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EXECUTIVE  SUMMARY 


Objective 


The  Collective  Protection  (COLPRO)  Business  Area  Manager  (BAM)  is 
continuously  faced  with  investment  decisions,  i.e.,  deciding  how  much  of  his  limited  Tech  Base 
funding  should  be  invested  in  which  technology  R&D  programs.  The  BAM  needs  a  method  to 
help  him  develop  and  examine  alternative  funding  strategies  for  different  technology  thrust  areas 
(e.g.  Air  Purification,  Shelter  Materials,  and  Critical  Components)  within  the  business  area  to 
determine  which  investment  portfolio  will  provide  the  highest  return. 

Method  and  Approach 

The  Decision  Analysis  Team  worked  with  the  COLPRO  BAM  and  subject  matter 
experts  (SMEs)  to  develop  resource  allocation  models  that  help  identify  the  combinations  of 
technologies  that  offer  the  most  efficient  use  of  a  finite  budget.  Three  software  packages  were 
used  to  support  the  process— DPLtm,  EXCEL™,  and  EQUITYtm. 

Using  DPL™  to  create  a  decision  tree  model,  each  high  potential  technology  was 
represented  as  a  separate  “investment  decision”,  i.e.,  whether  or  not  to  fund  a  technology,  and  at 
what  level.  Strategy  tables  were  then  created  to  identify  sets  of  technology  investment  decisions 
(Figure  1),  design  alternative  investment  strategies — no  investment,  low  investment,  moderate 
investment,  or  high  investment  (Figure  2),  and  evaluate  them  to  determine  the  expected  value  of 
each  strategy. 


For  each  technology,  a  panel  of  SMEs  estimated  the  costs  of  alternative 
investment  levels,  probabilities  of  success,  technology  benefit  levels,  and  overall  technology 
weights.  The  weights  reflect  each  technology’s  relative  importance,  and  are  based  on  potential 
performance  and  current  maturity  levels. 


No  Invest 


Low  Invest 


Mod  Invest 
\  High  Invest 


Figure  2.  Alternative  Investment  Levels 
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Figure  1 .  DPL  Strategy  Table 


(Figure  3). 


These  assessed  measures  were  compiled  into  an  EXCEL^*^  spreadsheet 
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Figure  3.  Example  Technology  Program  Estimates 


Results/Products: 

The  decision  tree  models  for  each  of  the  COLPRO  technology  thrust  areas 
produce  thousands  of  potential  combinations  of  investment  levels.  EQUITY™  is  a  resource 
allocation  software  tool  useful  for  examining  all  the  possible  combinations  of  the  technology 
investments  and  finding  those  combinations  that  provide  the  most  overall  benefit  for  any  given 
level  of  funding. 

In  EQUITY*''^,  a  Pareto  Diagram  is  a  graphical  representation  of  all  benefit-to- 
cost  combinations  (Figure  4).  In  this  diagram,  all  the  feasible  combinations  fall  within  the  green- 
shaded  area.  The  set  of  optimum  investment  strategies  is  represented  in  the  diagram  on  the 
upper  “frontier”  of  the  diagram.  These  are  the  best  strategies  across  a  range  of  different 
investment  funding  levels,  or  costs.  A  “best  value”  investment  strategy  is  located  at  the  “knee” 

of  the  frontier  curve.  Beyond  the  knee,  additional 
funding  provides  diminishing  returns. 

The  priority  order  for  funding  the 
technologies  and  the  funding  levels  are  listed  in  an 
“Order  of  Buy”,  showing  the  incremental  and 
cumulative  costs  and  benefits  for  the  technologies 
(Figure  5).  From  this  model  and  assessment 
exercise,  a  portfolio  of  technology  investments  that 
provides  the  best  use  of  limited  funding  can  be 
determined. 
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Figure  4.  Example  EQUITY  Pareto  Diagram 
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Figure  5.  Example  Order  of  Buy 


The  BAM  developed  three  initial  investment  models — one  for  each  of  his 
technology  thrust  areas.  Two  Working  Groups  comprised  of  COLPRO  SMEs  attempted  to 
validate  the  best  value  strategies  resulting  from  the  models  by  independently  developing  model 
parameters.  The  Working  Groups  restructured  the  models  and  provided  a  number  of  assessments 
that  were  significantly  different  from  the  BAM.  The  resulting  Working  Group  models  for  Air 
Purification  and  Shelters  were  not  able  to  validate  the  best  value  strategies  from  the  initial 
models. 
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PREFACE 


The  work  described  in  this  report  was  started  in  August  2002  and  completed  in 
September  2003. 

This  report  was  prepared  in  response  to  a  request  from  the  Collective  Protection 
Business  Area  Manager  (COLPRO  BAM)  to  extend  the  CP  Business  Area  Master  Planning 
Model  developed  during  the  COLPRO  Front  End  Analysis  and  Master  Planning  process  in  2001. 
It  is  an  annex  to  the  previously  published  FEA/MP  report. 

The  use  of  either  trade  or  manufacturers’  names  in  this  report  does  not  constitute 
an  official  endorsement  of  any  commercial  products.  This  report  may  not  be  cited  for  purposes 
of  advertisement. 

This  report  has  been  approved  for  public  release.  Registered  users  should  request 
additional  copies  from  the  Defense  Technical  Information  Center;  unregistered  users  should 
direct  such  requests  to  the  National  Technical  Information  Center. 
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TECHNOLOGY  INVESTMENT  STRATEGY  ANNEX, 
COLLECTIVE  PROTECTION  FRONT  END  ANALYSIS  AND 
MASTER  PLAN  REPORT 


1.  OVERVIEW 

The  Chemical  Biological  Collective  Protection  (CB  COLPRO)  Master  Plan 
Summary,  dated  February  2002,  stated;  “Key  objectives  of  the  Master  Plan  were  to  determine 
areas  of  technical  emphasis  within  the  COLPRO  business  area,  examine  funding  strategy 
alternatives,  and  determine  the  combinations  of  technologies  that  can  provide  the  most  effective 
use  of  a  finite  budget.  These  combinations  are  represented  by  variables  in  a  software  model. . . . 
The  model  allows  the  Business  Area  Manager  (BAM)  to  consider  a  number  of  strategy  variations 
to  test  the  robustness  of  the  investment  strategy  for  each  technology  category.  These  exercises 
can  demonstrate  where  it  may  be  desirable  to  make  less  risky  investments  and  accept  more 
modest  expected  returns.” 

This  Technology  Investment  Strategy  Annex  provides  conclusions  based  on  these 
exercises  and  is  the  final  step  in  the  COLPRO  Front  End  Analysis  (CP  FEA)  and  Master 
Planning  (CP  MP)  processes  for  2001-2002.  The  Technology  Investment  Strategies  were 
developed  in  a  series  of  meetings  between  the  CB  COLPRO  BAM  and  subject  matter  experts 
(SME)  from  January  2002  to  August  2002. 

The  CP  FEA  process  consisted  of  a  technical  assessment  on  all  viable  Air 
Purification  and  Shelter  Materials  and  Treatments  technologies.  The  result  of  the  assessment 
was  a  ranking  of  the  technologies  relative  to  five  application  areas.  The  rankings  were  based  on 
how  well  each  technology  performed  against  14  criteria  that  were  oriented  towards  Efficacy, 
Operational,  Logistical,  and  Safety,  Health,  and  Environment  considerations.  These  results 
indicated  each  technology’s  potential  at  satisfying  the  set  of  user  requirements  as  described  in  the 
various  program  Operational  Requirements  Documents  (ORD). 

The  CP  FEA  results  and  technology  rankings  were  used  to  generate  a  set  of  “high 
potential”  technologies.  The  CP  MP  process  then  evaluated  these  select  technologies  against 
other  considerations,  such  as  maturity  and  data  gaps,  to  determine  their  potential  to  transition 
into  viable  products.  The  first  product  of  this  process  was  the  identification  of  the  technologies 
that  will  be  emphasized  in  the  Tech  Base  Program,  and  when  they  may  be  available  to  transition 
into  acquisition  programs.  A  second  product  of  the  process  was  a  strategic  resource  allocation 
model  that  would  help  the  BAM  determine  how  to  invest  available  funding  in  the  COLPRO  area, 
both  at  the  technology  level  and  at  higher,  programmatic  levels.  The  CP  Master  Plan  process 
involves  four  steps: 

1 .  Define  the  COLPRO  business  area  program  framework; 

2.  Assess  COLPRO  high  potential  technologies,  in  terms  of:  maturity  and  data 
gaps/limitations,  technology  program/research  activities,  resource  profiles,  and 
technical  risk; 

3.  Prioritize  high  potential  technologies  and  establish  transition  timeframes; 

4.  Develop  planning  models  and  examine  alternative  program  strategies. 
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The  first  three  steps  were  documented  in  Section  8  of  the  Collective  Protection 
Front  End  Analysis  and  Master  Plan  Report,  dated  March  2002.  The  final  step  involves 
determining  areas  of  emphasis  within  the  COLPRO  Business  Area.  This  includes  determining 
investment  priorities  within  the  technology  thrust  areas,  the  technology  categories,  and  between 
the  various  technologies  themselves. 

Although  the  technologies  were  prioritized  in  the  CP  MP,  program  resources 
necessary  to  fund  6.2  development  for  all  the  high  priority  technologies  are  not  likely  to  be 
available.  The  objective  of  the  investment  strategy  exercises  was  to  examine  alternative  funding 
strategies  and  determine  the  mix  of  technologies  that  can  provide  the  most  effective  use  of 
limited  dollars.  A  key  assumption  was  that  a  mix  of  investments  in  a  range  of  high  value,  but 
potentially  redundant  technology  categories,  would  be  a  better  investment  strategy  than  investing 
solely  in  one  or  two  categories.  In  addition,  technologies  that  potentially  contribute  to  more  than 
one  acquisition  program  or  application  area  would  be  more  cost-effective  investments. 

To  accomplish  this  final  objective,  an  analytical  framework  was  constructed 
during  the  CP  MP  process.  The  framework  is  a  decision  tree  model  based  on  the  software 
package  DPL'^'^  (Applied  Decision  Analysis,  LLC).  The  decision  tree  let  the  BAM  choose  any 
set  of  technology  investment  decisions  that,  together,  comprise  an  investment  strategy,  and 
shows  expected  benefits  and  costs  resulting  from  the  strategy.  An  advantage  of  the  decision  tree 
is  the  ability  to  incorporate  uncertainty  by  examining  various  probabilities  of  success. 

Sensitivity  analysis  can  also  be  performed  to  see  how  varied  inputs,  such  as  expected  project 
costs,  affect  the  outcomes. 

The  process  used  to  develop  the  DPL^*^  decision  tree  model  is  described  in 
Section  8.5  of  the  Collective  Protection  Front  End  Analysis  and  Master  Plan  Report,  dated 
March  2002.  The  resulting  decision  tree  model  was  used  to  analyze  a  small  number  of 
investment  strategies  for  the  BAM.  That  model  showed  that  for  both  AP  and  Shelters,  the 
expected  benefits  from  technology  investment  reached  a  point  of  diminishing  return.  That  “knee 
of  the  curve”  suggested  that  a  “best  value”  strategy  existed  for  each  technology  thrust  area. 

However,  the  decision  tree  model  was  not  able  to  help  the  BAM  to  easily  identify 
the  best  strategy  to  recommend  among  the  thousands  of  potential  combinations  of  funding  levels 
for  each  technology  thrust  area.  For  this  answer,  resource  allocation  models  were  built  based  on 
the  decision  tree  framework. 

The  CB  COLPRO  BAM  met  with  members  of  the  Decision  Analysis  Team 
(DAT)  in  January  2002  to  develop  Technology  Investment  Strategies  for  each  of  the  three 
Technology  Thrust  Areas;  Air  Purification,  Shelter  Materials  and  Treatments,  and  Critical 
Components.  These  strategies  were  developed  using  “initial”  resource  allocation  models  that 
extended  the  decision  trees  discussed  in  the  Executive  Summary  for  assessing  the  various 
investment  priorities  and  allowed  the  BAM  to  explore  and  optimize  technology  investments  for 
any  given  level  of  resources.  These  initial  models  were  later  revised  by  Working  Groups 
consisting  of  the  BAM  and  selected  SMEs. 
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The  resource  allocation  models  were  developed  using  Microsoft  EXCEL™  and 
the  EQUITY^'^  software  package  (Enterprise  LSE,  Ltd.).  EQUITY™  is  a  commercial  modeling 
tool  used  for  a  wide  variety  of  investment  planning  applications  in  government  and  industry.  It 
uses  the  marginal  benefit-to-cost  ratio  of  each  investment  option  to  create  an  ordered  list  of 
investments  where  the  priority  order  of  investment  choices  does  not  change  if  the  resources 
increase  or  deerease.  The  software  allowed  the  BAM  and  DAT  to  meet  the  study  objective  of 
examining  alternative  funding  strategies  to  determine  the  combination  of  technologies  that  would 
efficiently  use  limited  funds. 


2.  TECHNOLOGY  INVESTMENT  STRATEGIES 

2.1  Elements  of  the  Strategy  Models. 

There  are  four  major  elements  to  an  EQUITY™  investment  strategy  model:  1) 
the  structure  of  investment  areas  and  levels,  2)  expected  payoffs,  or  relative  returns  on 
investment  levels,  3)  probabilities  of  each  potential  outcome,  and  4)  priorities,  or  importance 
weights,  among  the  various  investment  areas. 

2.1.1  Structure. 

The  basic  stmcture  of  the  strategy  model  resembles  a  matrix.  The  technology  investment 
categories  are  listed  down  the  left-hand  side  of  the  matrix,  and  the  potential  levels  of  investment 
are  listed  across  the  top  (Figure  1). 


1  [  j]  EQUITY  for  Windows 

-  [  c:Vnydocu~1\other\equity\cpmast“1\cp_ap_03.eqw  ]  BEIE3| 

:  File  Edit  View  Configure 

clipboard  Anali^sis 

Help 

L£|lii'|'inil'ii!il  ISi  1 

it, 

1 

2 

3 

A  5 

1  Tech  A 

if 

None 

Lo\v 

Mod 

High  ^Maximum 

!■ 

1?  Tech  B 

■  None 

Low 

Mod 

High  ^Maximum 

1  Tech  C 

1 

.  None 

Low 

Mod 

High  JMaximum 

1  Tech  D 

1  -  .  . 

None 

Low 

Mod 

High  pMaximum 

1  Tech  E 

None 

Low 

Mod 

. . . . „i 

High  ^Maximum 

1 

Figure  1 .  Technology  Investment  Strategy  Model 


The  potential  levels  of  investment,  or  the  amount  of  resourees  spent,  increase 
going  from  left  to  right.  A  “None”  level  (Level  1)  is  included  to  show  that  it  is  feasible  not  to 
invest  in  a  technology  category  at  all.  A  “Maximum”  level  of  investment  (Level  5)  is  included  to 
show  that  there  is  a  theoretical  amount  of  resources  that  could  mitigate  all  risk  in  a  technology 
category  and  guarantee  success,  but  it  would  be  prohibitively  expensive.  A  Moderate  level  of 
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investment  was  developed  for  each  technology  category  based  on  the  “nominal”  or  baseline  cost 
estimates  from  the  Master  Plan.  Increasing  or  decreasing  the  baseline  levels,  generally  by  50% 
each  way,  also  developed  Low  and  High  levels  of  investment. 

2.1.2  Payoffs. 

The  expected  results,  or  payoffs,  from  a  technology  investment  are  the  next 
elements  of  the  model.  The  initial  models  used  three  potential  outcomes,  defined  as: 

•  Minimal  Success:  The  research  goals  are  not  achieved,  but  there  may  be  some 
residual  contribution  to  the  overall  program; 

•  Partial  Success:  Many  program  objectives  are  achieved,  such  that  the  benefits 
achieved  are  somewhat  more  than  halfway  between  the  Minimal  and  Complete  levels; 

•  Complete  Success:  All,  or  nearly  all,  program  objectives  are  achieved. 

In  the  models,  each  potential  outcome  has  an  expected  relative  benefit  to  the 
program.  A  relative  benefit  score  of  1 00  is  assigned  to  Complete  Success.  A  relative  benefit  of 
0  would  result  from  no  investment  in  the  technology.  The  Minimal  and  Partial  results  represent 
the  relative  value  of  intermediate  degree  of  success  outcomes  on  the  0-100  interval  scale. 

For  the  initial  models,  the  BAM  assessed  the  relative  values  shown  below  in 

Table  1. 


Table  1 .  Initial  Model  Payo: 

ffs 

Complete  Success 

100 

Partial  Success 

55 

Minimal  Success 

10 

No  Success 

0 

The  BAM  assumed  that  relatively  low  value  would  result  from  any  development 
effort  that  only  had  minimal  success — about  10%  of  the  total  value  of  a  completely  successful 
effort — and  a  partial  success  would  achieve  somewhat  more  than  50%  of  the  total  possible  value. 
With  these  assessments,  the  BAM  expressed  a  neutral  “risk  preference”  toward  project  success  in 
that,  in  his  judgment,  increases  in  project  value  are  commensurate  with  increases  in  degree  of 
success. 

2.1.3  Probabilities. 

The  BAM  assumed  that  the  probability  of  each  potential  outcome  in  the  previous 
section  would  vary  depending  on  the  amount  of  resources  invested.  In  other  words,  he  would  be 
able  to  increase  the  chances  of  success  by  investing  more  resources  in  a  technology  category. 

The  BAM  developed  initial  models  with  four  feasible  levels  of  investment — ^None,  Low, 
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Moderate,  and  High — and  assessed  the  probabilities  of  success  across  the  outcomes  using  a 
matrix  like  the  one  shown  below  (Table  2). 


Table  2.  Example  of  Probabilities  of  Success 


Program  Outcome  Probability  of  Success 

Investment  Level 

Minimal 

Partial 

Complete 

Low 

.5 

.4 

.1 

Moderate 

.2 

.6 

.2 

High 

.1 

.5 

.4 

The  probabilities  along  each  row  in  the  table  above  must  sum  to  1.0.  An 
investment  level  of  “None”  always  results  in  No  Success.  In  contrast  to  the  single  set  of  payoff 
values  that  the  BAM  applied  to  all  technology  categories  (Table  1),  the  BAM  assessed  a  different 
pattern  of  probabilities  for  each  technology  category.  These  assessed  probability  sets  are  shown 
in  Tables  3, 4,  and  5.  Although  the  BAM  assessed  equivalent  payoff  values  for  eaeh  technical 
category,  the  probabilities  assigned  by  the  BAM  varied  for  each  technical  category. 


Table  3.  Air  Purification  Initial  Model  Inputs 


Technologies 

Program  Outcome  Probability  of 
Success 

Probability 

Adjusted 

Benefits 

Technology 

Weight 

Minimal 

Partial 

Complete 

Total 

Prob. 

02  Supply 

— on — 

0.1b 

O.Ob 

Uiisi 

- TTT^ 

21% 

on 

0:20 

jUyjl 

;5T30 

oo 

0.40 

00 

mfjji 

65.00 

Technology  Benefit 

10 

55 

100 

CatOx 

Sb.UUU 

(J.bO 

U.bO 

- 00 - 

HBUSfl 

32750 

19% 

|yJy^^l|| 

0.15 

rEE] 

43. 75 

KIESSfifil 

03 

0.40 

O'^S 

USiil 

50.50 

Technology  Benefit 

10 

55 

100 

WStiA 

Regen 

0.1^5 

. O'.'i'O' . 

0.0  b 

23. bO 

17% 

57,700 

- - 

imyji 

0.15 

Uiifl 

43.75 

igmjgtii 

on 

0.40 

00 

BfiiSi 

55.00 

Technology  Benefit 

10 

55 

100 

Noncarbon  Materials 

— (j.yu " 

u.ifi) 

'"O'.  05""’"“ 

mmm 

25.75 

8% 

KiMiBsliSH 

0.4b 

lOO 

crm 

ttUUAI 

- ^ - 

■SUi 

00 

^2S1SI 

55.00 

Technology  Benefit 

10 

55 

100 

Nonmembrane  Filters 

51,bUU 

— nm — 

U.2b 

U.Ub 

Uiiil 

25.75 

8% 

— (J-7C 

mygjl 

0.30 

50.50 

K&S&IS 

on 

0.30 

USISI 

WBilUi 

7700 

'  ,  .  Technology  Benefit 

10 

55 

100 

Engineered  Composites 

■IBiW 

U.bb 

U.4U 

U.Ub 

370D 

6% 

HSflSSiSH 

U.^b 

U.2b 

bb.OU 

OS 

B2m3i 

IRO 

mijjii 

S5:3S 

Technology  Benefit 

10 

55 

100 

Fiber  Filter  T reatments 

bl.bUU 

U.bU 

u.bb 

u.bb 

30. 2b 

8% 

U.2b 

gi22sH 

U.2b 

■flS^ 

bb.OU 

tiiWgIIF1..iij 

nyjom 

on 

0.35 

ERS 

UflSi 

5f:7s 

Technology  Benefit 

100 

WilSI 

Activated  Carbon 

KIBimB 

oo 

0.35 

0.15 

3n:'25 

13% 

OS 

0.50 

0:5s 

55.00 

0.15 

ERO 

0.45 

68. 5o 

Technology  Benefit 

10 

55 

100 

5 


Table  4.  Shelter  Materials  Initial  Model  Inputs 


Techngla 


Heterogeneous 


Homogeneous 


Multilayer 


Program  Outcome  Probability  of 
Success 

Minimal 

Partial 

Complete 

Total 

Prob. 

Probability  I 
Adjusted 
Benefits 


Technology 

Weight 


.  ‘  technology  Benefit 


:  . ; '  Technology  Benefit 


Semipermeable 


■MiliiiiiMtiiilli 


Selectively  Permeable 


High  Surface  Tension 


aiiissi 


,  ,  Technology  Benefit 


2.1.4  Priorities. 

The  final  model  element — priorities — is  a  relative  weight  on  the  technology 
categories  that  indicates  the  importance,  or  impact  on  the  Business  Area,  of  transitioning  each 
one.  While  the  value  of  success  for  each  technology  category  was  the  same,  some  technologies, 
if  successfully  transitioned,  would  be  preferred  to  other  technologies.  Therefore  the  BAM 
assigned  a  higher  weight  to  the  0-100  scales  of  higher  priority  technologies. 

The  BAM  used  a  combination  of  techniques  to  prioritize  the  categories  in  each  of 
the  three  technology  thrust  areas,  including  an  ordinal  ranking  technique,  a  pairwise  comparison 
technique,  and  an  anchoring  and  adjusting  technique.  In  the  ordinal  ranking  technique,  called 
Simplified  Multi-Attribute  Rating  Technique — Ranking  (SMARTER),  the  BAM  assessed 
weights  by  listing  the  rank  order  of  importance  of  each  technology  category.  Decision  support 
software  (Logical  Decisions  for  Windows"^*^)  uses  the  importance  ordering  to  compute  a  set  of 
implied  weights.  The  set  of  weights  represents  a  center  of  mass  of  all  the  possible  sets  of 
weights  consistent  with  the  ordering  using  a  “centroid”  algorithm. 
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Table  5.  Critical  Components  Initial  Model  Inputs 


Technologies 

Program  Outcome  Probability  of 
Success 

Probability 

Adjusted 

Benefits 

Technology 

Weight 

Minimal 

Partial 

Complete 

Total 

Prob. 

Closures  Seams  and 
Seals 

mUegiiyjig 

■SSI 

■muM 

41 .5o 

26% 

KS9 

USiSI 

65. Oo 

■KMi 

O.iO 

wmm 

0:60 

HSiSI 

Technology  Benefit 

10 

55 

100 

Airbeam 

■ilffilSI 

0.30 

0.30 

13% 

trro 

USiSI 

64.00 

■HB 

■SISISliB 

0.10 

030 

USiSI 

7TOO 

Technology  Benefit 

10 

55 

100 

Tension  Frame  Fabric 

— ^ 

USiSi 

0730 

3% 

mSKBSmSSaSE^SISSK 

KUSSBB 

030 

USiSI 

BSKi: 

■SBfilSB 

mtmm 

0.55 

■Kllf 

UsUSH 

Technology  Benefit 

10 

55 

100 

Composite  Frame 
Hinge 

— gEiBVW 

■SfilSH 

o:ttj 

USiSI 

3730 

7% 

0.40 

0.40 

usisi 

BHEUBimiliB 

fgjjgjgg 

USiSI 

UsIsISSfll 

Technology  Benefit 

10 

55 

100 

USiSI 

Threat  Mitig  Methods 

■aESKiflaiiai 

KSUH 

— ^ — 

50.50 

10% 

0.15 

MiicTM 

0.55 

mmm 

i— E'nr  ■ii.iii«r»M 

0.05 

KIBsH 

070 

Usisi 

Technology  Benefit 

10 

55 

100 

USiSI 

Airlocks/  Barriers/Doors 

imiM 

BiKHH 

WISSSKM 

— ^ — 

43. /5 

19% 

KEISH 

0:^0 

USiSI 

USBilSU 

mmm 

0.10 

USiSI 

UsUSH 

Technology  Benefit 

10 

55 

100 

H^2i8l 

Integrated  Power  ECU 

Hpis 

imgjjjii 

BBSH 

U.15 

3730 

16% 

K&fiSlSH 

0.30 

HSfitil 

U.as 

USiSI 

miuM 

MSSlM 

WUSM 

USiSI 

UU3SU 

Technology  Benefit 

10 

55 

100 

Energy  Eff  Materials 

0.6 

0.3 

0.1 

UE9BSU 

6% 

'S  il  L.  'lli ' '  if '  . .  ?  '!i; : 

0.4 

MiCTi 

0.3 

MTiia 

50.50 

ipfpia^sitTir 

0.3 

■EEH 

0.4 

mwiim 

59.50 

Technology  Benefit 

10 

55 

100 

In  the  pairwise  comparison  technique,  called  the  Analytic  Hierarchy  Process 
(AHP),  the  BAM  defined  the  weights  by  assessing  ratios  of  the  technology  category 
importance — an  importance  ratio  for  each  possible  pair.  In  the  AHP  method,  an  approach  based 
on  linear  algebra  is  used  to  compute  a  "best  fit"  set  of  weights  based  on  the  ratios  entered. 

In  the  anchoring  and  adjusting  technique,  called  Simplified  Multi-Attribute  Rating 
Technique  (SMART),  the  BAM  defined  the  weights  by  entering  the  relative  importances  in  the 
form  of  "swing  weights".  Swing  weights  describe  the  relative  importance  of  "swinging"  a 
technology  from  its  least  preferred  to  its  most  preferred  level.  He  assigned  a  weight  of  100  to  the 
technology  that  is  most  important  to  swing  to  its  most  preferred  level.  He  then  assigned  lower 
weights  to  the  other  technologies  based  on  the  relative  importance  of  swinging  them  vs.  the  most 
important  technology.  The  assessed  weights  are  shown  in  Tables  3, 4,  and  5  in  the  right-most 
column.  The  BAM  used  the  weights  computed  by  Logical  Decisions  for  Windows™;  each 
method  refined  the  weights  computed  by  the  previous  method. 
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2.2 


Initial  Air  Purification  Technology  Investment  Model. 


The  first  of  the  three  initial  models  was  developed  for  the  Air  Purification 
technology  thrust  area.  The  CP  FEA  identified  eight  high  potential  technology  categories  and 
the  Master  Plan  Report  provides  detailed  descriptions  of  them.  The  model  inputs  are  shown  in 
Table  3.  The  numbers  in  the  Probability  Adjusted  Benefits  (PAB)  column  were  obtained  by 
multiplying  the  Program  Outcome  Probability  of  Success  for  the  Minimal,  Partial,  and  Complete 
levels  by  the  respective  Technology  Benefit  levels  and  then  by  summing  all  of  the  products  for 
each  of  the  investment  levels.  For  example,  to  calculate  the  PAB  for  the  Low  Investment  level 
of  02  Supply  (21.25)  one  would  do  the  following:  (0.80*10)  +  (0.15*55)  +  (0.05*100). 

Each  technology  category  was  treated  as  a  separate  investment  area  (row)  within 
the  model  (Figure  2).  For  each  potential  investment  level  along  a  row,  the  cumulative  costs  and 
probability-adjusted  benefits  were  assigned  (from  Table  3). 
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Figure  2.  AP  Initial  Technology  Investment  Model 


The  model  produces  a  Pareto  diagram  of  all  possible  combinations  of  funding  levels 
(390,625)  EQUITY  normalizes  all  benefits  in  a  model  so  that  the  total  possible  is  always  a 
maximum  of  1000  relative  benefit  points  (Figure  3).  However,  even  if  the  highest  investment 
was  made  on  all  technologies  (total  of  $64.2M)  the  BAM  could  not  expect  to  achieve  more  than 
about  60%  of  the  maximum  benefit  because  of  uncertainty  about  the  success  of  the  development 
efforts  (593  out  of  1000  points). 

The  green  shaded  area  at  the  top  of  the  diagram  shows  where  these  “infeasible” 
strategies  lie.  Infeasible  strategies  include  at  least  one  maximum  level  (Level  5),  which  assumes 
a  100%  guaranteed  complete  success.  The  BAM  did  not  consider  this  possibility  to  be  feasible. 
The  set  of  optimum  frontier  strategies,  therefore,  lies  along  the  top  edge  of  the  yellow  shaded 
area  of  feasible  strategies. 
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Because  the  frontier  curve  is  quite  smooth,  there  is  no  “knee”  and  so  it  is  not  clear 
where  the  “best  value”  strategy  lies.  The  “F”  point  along  the  frontier  shows  a  typical  best  value 
strategy,  where  for  a  little  less  than  50%  of  the  total  cost,  or  $28.9  M,  the  program  could  achieve 
382  points,  or  64%  of  the  feasible  benefit.  This  suggested  best  value  strategy,  called  the  Frontier 
Package,  is  shown  in  Figure  4. 


1  ■  Frontier  package  IH 

IBB 

:  Frontier  Pack  #12 

Preference  Values 

COSTS 

BENEFITS 

TECH 

LEVEL 

Cost 

Benefit 

Total 

Total 

1  02Supp!y 

>  2  Lov/ 

4500 

4500 

45 

45 

!  2  CatOx 

2  Lov'/ 

5000 

5000 

62 

62 

;  3  Regen 

2  Low 

3850 

3850 

40 

40 

i  4  NonCarbon 

2  Low 

1750 

1750 

21 

21 

i  5  NonMemFilt 

<  4  High 

4500 

4500 

62 

62 

^  6  EngComp 

2  Low 

1800 

1800 

20 

20 

^  7  FibFilTreat 

3  Mod 

3000 

3000 

44 

44 

8  ActCarbon 

4  High 

4500 

4500 

89 

89 

'  Frontier  package 

28900 

28900 

382 

382 

;  Next  Package  #13 

33400 

33400 

424 

424 

Previous  Pack  #1 1 

27400 

27400 

367 

367 

Figure  4.  AP  Best  Value  Strategy 
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For  each  technology  category,  the  recommended  level  of  investment  is  shown  in 
the  LEVEL  column.  The  RED  arrow  indicates  the  level  that  would  be  increased  next  if 
additional  funds  were  invested  (the  next  “buy”).  The  RED  Next  Package  #13  shows  the  total 
cost  and  benefit  of  that  “next”  strategy.  The  BLUE  arrow  indicates  the  level  that  would  be 
reduced  next  if  funds  were  cut  (the  next  “sell”).  The  BLUE  Previous  Package  #11  shows  the 
total  cost  and  benefit  of  that  “previous”  strategy. 

The  COSTS  and  BENEFITS  columns  show  the  costs  and  benefits  for  each 
technology  category  at  the  recommended  levels.  The  Total  columns  are  the  same  as  the  Cost  and 
Benefit  columns  because  there  are  only  one  type  of  cost  and  one  type  of  benefit  in  the  model. 


The  complete  Order  of  Buy  is  shown  in  Figure  5.  This  is  the  investment  list  that 
the  BAM  should  follow  in  order  to  maximize  the  value  of  his  investments  at  any  cumulative 
level  of  funding.  The  best  value  strategy  is  shown  at  increment  #12. 
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48 
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#4 

8- 
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3 
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20 
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#5 

7- 
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3 
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20 
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m 

2- 
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2 
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62 
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m 

4- 
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2 
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1750 
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21 
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d 

8~ 

ActCarbon 

4 
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18 
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m 

6- 

EngComp 

2 

Low 

1800 

19050 

20 

283 

#10 

3- 

Regen 

2 

Low 

3850 

22900 

40 

322 

#11 

1  - 

02Supply 

2 

Low 

4500 

27400 

45 

367 

#12 

5- 

NonMemFilt 

4 

High 

1500 

28900  , 

14 

382 

#13 

1  - 

Q2Supply 

3 

,  Mod 

4500 

';;;''3340|v.;:'.: 

424 

#14 

3- 

Regen 

3 

Mod 

3850 

37250  :  . 

459 

#16 

B* 

EngComp 

. 3:’ 

Mod 

1800 

H'#p50'g:^:- 

472 

#16 

4  - 

NonCarbon 

.  '  3/ 

Mod 

1750 

40800  "  : 

13. -■■■■.■ 

485 

#17 

1  - 

02Supply 

4 

High 

4500 

45300  ^ 

513 

#18 

A-’ 

NonCarbon 

4 

High 

1750 

4705Q 

11 

524 

#19 

7- 

FibFilTreat 

4 

High 

1500 

485^ 

533 

#20 

3- 

Regen 

4 

High 

3850 

52400 

552 

#21 

2- 

CatOx 

3 

Mod 

5000 

67400 

21  , 

573 

#22 

6- 

EngComp 

4 

High 

1800 

59200 

'  ' 

580 

#23 

2- 

CatOx 

4 

High 

5000 

64200 

13 

593 

Figure  5.  AP  Initial  Order  of  Buy 


Order  of  Buy 

COSTS; 

BENEFITS 

LEVEL 

INC  CUM  V 

INC: 

CUM 

1 

None 

0 

0 

1 

None 

0 

0 

d 

0 

1 

None 

0 

0 

0 

0 

1 

None 

0 

0 

0 

0 

1 

None 

0 

0 

0 

0 

1 

None 

0 

0 

0 

0 

1 

None 

0 

0 

0 

0 

1 

None 

0 

0 

0 

0 
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Initial  Shelter  Materials  and  Treatments  Technology  Investment  Model. 

The  second  model  was  developed  for  the  Shelter  Materials  and  Treatments  technology  thrust 
area.  The  CP  FEA  identified  seven  high  potential  technologies  in  three  technology  categories 
(Table  4). 

Each  technology  was  treated  as  a  separate  investment  area  (row)  within  the  model 
(Figure  6).  For  each  potential  investment  level  along  a  row,  the  cumulative  costs  and 
probability-adjusted  benefits  were  assigned  (from  Table  4). 


1  [  3  EQUITY  for  Windows 

-  [  c:Vnydocu~1VDther\equity\cpmdst~1\cp_ 

sm_02.eqw]  BSDl 

_ _ _ ~  I 

File  Edit  View  Configure  ' 

Clipboard  Analysis 

r\ 

Help 

\d\^\ 

1; 

1  Heterog 

1 

1 

None 

2 

Low 

3 

Mod 

A 

High 

5 

|Maximufn 

1  Homog 

1: 

None 

Low 

Mod 

High 

^Maximum 

f  Multil 

I' 

None  1 

i 

Low 

Mod 

High 

|Maximunn 

f 

1  Semiperm 

None 

Low 

Mod 

High 

|Maximum 

1  SeiecPerm 

None 

Low 

Mod 

High 

f  Maximum 

1  HiSuiTen 

None 

Low 

Mod 

High 

1  Maximum 

1  ActReact 

i'' 

None 

Low 

j  Mod 

High 

1  Maximum 

Figure  6.  SM  Technology  Investment  Model 


The  model  produces  a  Pareto  diagram  of  all  possible  combinations  of  funding 
levels  with  a  maximum  of  1000  relative  benefit  points  (Figure  7).  However,  even  if  the  highest 
investment  was  made  on  all  technologies  (total  of  $29  M)  the  BAM  could  not  expect  to  achieve 
more  than  about  70%  of  the  maximum  benefit  because  of  uncertainty  about  the  success  of  the 
development  efforts  (693  out  of  1000  points). 


Figure  7.  SM  Pareto  Diagram 
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The  green  shaded  area  at  the  top  of  the  diagram  shows  where  these  “infeasible” 
strategies  lie.  (See  the  previous  section  for  an  explanation  of  infeasible  strategies.)  The  set  of 
optimum  frontier  strategies,  therefore,  lies  along  the  top  edge  of  the  yellow  shaded  area  of 
feasible  strategies. 

In  this  technology  thrust  area,  there  appears  to  be  a  clear  “knee”  on  the  frontier 
curve.  The  “F”  point  along  the  frontier  shows  a  best  value  strategy,  where  for  a  little  less  than 
20%  of  the  total  cost,  or  $5  M,  the  program  could  achieve  45 1  points,  or  65%  of  the  feasible 
benefit.  This  suggested  best  value  strategy  is  shown  in  Figure  8. 


1  ■  Frontier  package 

IBB 

Frontier  Pack  #5 

Preference  Values 

COSTS 

BENEFITS 

TECH 

LEVEL 

Cost 

Benefit 

Total 

Total 

1  Heterog 

2  Low 

1000 

1000 

106 

106 

2  Homog 

<  2  Low 

1000 

1000 

45 

45 

3  Multil 

2  Low 

1000 

1000 

148 

148 

4  Semiperm 

2  Low 

1000 

1000 

93 

93 

5  SelecPerm 

2  Low 

1000 

1000 

59 

59 

;  6  HiSurTen 

>  1  None 

0 

0 

0 

7  ActReact 

1  None 

0 

0 

0 

Frontier  package 

5000 

5000 

451 

451 

Next  Package 

6500 

6500 

482 

482 

Previous  Pack  M 

4000 

4000 

406 

Figure  8.  SM  Best  Value  Strategy 


For  each  technology  category,  the  recommended  level  of  investment  is  shown  in 
the  LEVEL  column.  The  RED  arrow  indicates  the  level  that  would  be  increased  next  if 
additional  funds  were  invested  (the  next  “buy”).  The  RED  Next  Package  #6  shows  the  total  cost 
and  benefit  of  that  “next”  strategy.  The  BLUE  arrow  indicates  the  level  that  would  be  reduced 
next  if  funds  were  cut  (the  next  “sell”).  The  BLUE  Previous  Package  #4  shows  the  total  cost  and 
benefit  of  that  “previous”  strategy. 

The  COSTS  and  BENEFITS  columns  show  the  costs  and  benefits  for  each 
technology  category  at  the  recommended  levels.  The  Total  columns  are  the  same  as  the  Cost  and 
Benefit  columns  because  there  are  only  one  type  of  cost  and  one  type  of  benefit  in  the  model. 

The  complete  Order  of  Buy  for  Shelter  Materials  is  shown  in  Figure  9.  This  is  the 
investment  list  that  the  BAM  should  follow  in  order  to  maximize  the  value  of  his  investments  at 
any  cumulative  level  of  funding.  The  best  value  strategy  is  shown  at  increment  #5. 
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■  Order  of  Buy 

ees 

Order  of  Buy 

COSTS 

BENEFITS 

TECH 

LEVEL 

INC 

CUM 

INC 

CUM  ^ 

1  - 

Heterog 

1 

None 

0 

0 

0 

0 

m 

2- 

Homog 

1 

None 

0 

0 

0 

0 

m 

3- 

Multil 

1 

None 

0 

0 

0 

0 

4- 

Semiperm 

1 

None 

0 

0 

0 

0 

m 

5- 

SelecPerm 

1 

None 

0 

0 

0 

0 

m 

6- 

HiSurTen 

1 

None 

0 

0 

0 

0 

]  m 

7- 

ActReact 

1 

None 

0 

0 

0 

0 

r  #1 

3- 

Multil 

2 

Low 

1000 

1000 

148 

148 

#2 

1  - 

Heterog 

2 

Low 

1000 

2000 

106 

254 

#3 

4- 

Semiperm 

2 

Low 

1000 

3000 

93 

347 

#4 

5- 

SelecPerm 

2 

Low 

1000 

4000 

59 

406 

#5 

2- 

Homog 

2 

Low 

1000 

5000 

45 

451 

m 

6- 

HiSurTen 

2 

Low 

1500 

6500 

31 

482 

iff 

7- 

ActReact 

2 

Low 

1500 

8000 

31 

513 

m 

3- 

Multil 

4 

High 

2000 

10000 

32 

545 

:  *9 

5- 

SelecPerm 

3 

Mod 

1000 

11000 

16 

561 

#10 

1  - 

Heterog 

4 

High 

2000 

13000 

25 

587 

#11 

6- 

HiSurTen 

3 

Mod 

1500 

14500 

18 

605 

#12 

4- 

Semiperm 

3 

Mod 

1000 

15500 

9 

614 

#13 

7- 

ActReact 

3 

Mod 

1500 

17000 

13 

628 

#14 

7- 

ActReact 

4 

High 

3000 

20000 

22 

649 

#15 

4- 

Semiperm 

4 

High 

2000 

22000 

13 

662 

#16 

5- 

SelecPerm 

4 

High 

2000 

24000 

12 

674 

#17 

2- 

Homog 

4 

High 

2000 

26000 

11 

685 

#18 

6- 

HiSurTen 

4 

High 

3000 

29000 

8 

693 

Figure  9.  SM  Order  of  Buy 


2.4  Initial  Critical  Components  Technology  Investment  Model. 

The  third  initial  model  was  developed  for  the  Critical  Components  technology 
thrust  area.  The  CP  FEA  identified  8  high  potential  technology  categories  (Table  5). 

Each  technology  category  was  treated  as  a  separate  investment  area  (row)  within 
the  model  (Figure  10).  For  each  potential  investment  level  along  a  row,  the  cumulative  costs  and 
probability-adjusted  benefits  were  assigned  (from  Table  5). 
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None 
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Mod 
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. . . . . J 
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j  ThrealMit 
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j 

Mod 
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.Maximum 
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[t  .  ■ 
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Figure  10.  Critical  Components  Technology  Investment  Model 


The  model  produces  a  Pareto  diagram  of  all  possible  combinations  of  funding 
levels  with  a  maximum  of  1000  relative  benefit  points  (Figure  1 1).  However,  even  if  the  highest 
investment  was  made  on  all  technologies  (total  of  $43  M)  the  BAM  could  not  expect  to  achieve 
more  than  about  73%  of  the  maximum  benefit  because  of  uncertainty  about  the  success  of  the 
develop  efforts  (73 1  out  of  1000  points). 


Figure  1 1 .  Critical  Components  Pareto  Diagram 
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The  green  shaded  area  at  the  top  of  the  diagram  shows  where  these  “infeasible” 
strategies  lie.  The  set  of  optimum  frontier  strategies,  therefore,  lies  along  the  top  edge  of  the 
yellow  shaded  area  of  feasible  strategies. 

In  this  technology  thrust  area,  there  appears  to  be  a  clear  “knee”  on  the  frontier 
curve.  The  “F”  point  along  the  frontier  shows  a  best  value  strategy,  where  for  a  little  less  than 
20%  of  the  total  cost,  or  $8.5  M,  the  program  could  achieve  409  points,  or  55%  of  the  feasible 
benefit.  This  suggested  best  value  strategy  is  shown  in  Figure  12. 


I  M  Frontier  package 

BBID| 

:  Frontier  Pack  W 

Preference  Values 

COSTS 

BENEFITS 

i  TECH 

LEVEL 

Cost 

Benefit 

Total 

Total 

i  1  Closures 

2  Low 

3000 

3000 

108 

108 

:  2  Airbeam 

2  Low 

1000 

1000 

72 

72 

1 3  TenFrame 

<  2  Low 

500 

500 

11 

11 

4  CompPan 

2  Low 

1000 

1000 

26 

26 

5  ThreatMit 

2  Low 

1000 

1000 

51 

51 

6  Airlocks 

2  Low 

1000 

1000 

83 

83 

i  7  PowerECU 

2  Low 

1000 

1000 

59 

59 

8  EnergEff 

>  1  None 

0 

0 

0 

0 

Frontier  package 

8500 

8500 

409 

409 

Next  Package  #8 

9500 

9500 

429 

429 

Previous  Pack 

8000 

8000 

398 

398 

Figure  12.  Critical  Components  Best  Value  Strategy 


For  each  technology  category,  the  recommended  level  of  investment  is  shown  in 
the  LEVEL  column.  The  RED  arrow  indicates  the  level  that  would  be  increased  next  if 
additional  funds  were  invested  (the  next  “buy”).  The  RED  Next  Package  #8  shows  the  total  cost 
and  benefit  of  that  “next”  strategy.  The  BLUE  arrow  indicates  the  level  that  would  be  reduced 
next  if  funds  were  cut  (the  next  “sell”).  The  BLUE  Previous  Package  #6  shows  the  total  cost  and 
benefit  of  that  “previous”  strategy. 

The  COSTS  and  BENEFITS  columns  show  the  costs  and  benefits  for  each 
technology  category  at  the  recommended  levels.  The  Total  columns  are  the  same  as  the  Cost  and 
Benefit  columns  because  there  are  only  one  type  of  cost  and  one  type  of  benefit  in  the  model. 

The  complete  Order  of  Buy  for  Critical  Components  is  shown  in  Figure  13. 

This  is  the  investment  list  that  the  BAM  should  follow  in  order  to  maximize  the  value  of 
his  investments  at  any  cumulative  level  of  funding.  The  best  value  strategy  is  shown  at 
increment  #7. 
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X. 


Order  of  Buy 


COSTS  BENEFITS 


TECH 

LEVEL 

INC 

CUM 

INC 

CUM 

m 

1  - 

Closures 

1 

None 

0 

0 

0 

0 

m 

2- 

Airbeam 
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None 

0 

0 

0 

0 

m 

3- 

TenFrame 

1 

None 

0 

0 

0 

0 

m 

4- 

CompPan 

1 

None 

0 

0 

0 

0 

m 

6> 

ThreatMit 

1 

None 

0 

0 

0 

0 

m 

6- 

Airlocks 

1 

None 

0 

0 

0 

0 

m 

7* 

PowerECU 

1 

None 

0 

0 

0 

0 

m 

8- 

EnergEff 

1 

None 

0 

0 

0 

0 

#1 

6- 

Airlocks 

2 

Low 

1000 

1000 

83 

83 

#2 

2- 

Airbeam 

2 

Low 

1000 

2000 

72 

155 

#3 

7 . 

PowerECU 

2 

Low 

1000 

3000 

59 

214 

m 

5- 

ThreatMit 

2 

Low 

1000 

4000 

51 

264 

m 

1  - 

Closures 

2 

Low 

3000 

7000 

108 

372 

m 

4- 

CompPan 

2 

Low 

1000 

8000 

26 

398 

3- 

TenFrame 

2 

Low 

500 

8500 

11 

409 

#3 

8- 

EnergEff 

2 

Low 

1000 

9500 

20 

429 

m 

7- 

PowerECU 

3 

Mod 

2000 

11500 

32 

461 

#10 

6- 

Airlocks 

3 

Mod 

2000 

13500 

30 

491 

#11 

6- 

Airlocks 

4 

High 

3000 

16500 

43 

534 

#12 

1  - 

Closures 

3 

Mod 

3000 

19500 

35 

569 

*13 

1  - 

Closures 

4 

High 

5000 

24500 

58 

627 

#14 

5- 

ThreatMit 

3 

Mod 

2000 

26500 

23 

650 

#15 

2- 

Airbeam 

3 

Mod 

1500 

28000 

12 

662 

*16 

2- 

Airbeam 

4 

High 

1500 

29500 

12 

673 

#17 

3- 

TenFrame 

3 

Mod 

500 

30000 

3 

677 

#18 

4- 

CompPan 

3 

Mod 

1000 

31000 

6 

683 

*19 

4- 

CompPan 

4 

High 

1000 

32000 

6 

689 

#20 

5- 

ThreatMit 

4 

High 

2000 

34000 

11 

701 

#21 

8- 

EnergEff 

3 

Mod 

2000 

36000 

11 

711 

#22 

7- 

PowerECU 

4 

High 

3000 

39000 

11 

722 

^3 

3  - 

TenFrame 

4 

High 

1000 

40000 

3 

726 

#24 

8- 

EnergEff 

4 

High 

3000 

43000 

5 

731 

Figure  13.  Critical  Components  Order  of  Buy 


3.  WORKING  GROUP  VALIDATION  SESSIONS 

The  COLPRO  Master  Planning  Working  Groups  (WG)  met  on  August  14  (Air 
Purification)  and  August  1 5,  2002  (Shelters)  at  the  Edgewood  Chemical  and  Biological  Center, 
Aberdeen  Proving  Ground,  MD.  Each  group  consisted  of  the  BAM  and  6  to  8  subject  matter 
experts.  The  objective  of  the  meetings  was  to  validate  the  initial  models  developed  by  the  BAM. 
See  Appendix  B  for  the  list  of  participating  SMEs. 
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3.1 


Air  Purification. 


Before  making  any  assessments,  the  Air  Purification  Working  Group  revised  the 
structure  of  the  initial  model  in  order  to  develop  the  technology  investment  strategy  at  a  higher 
level.  The  intent  was  to  simplify  the  model  for  resource  allocation  purposes. 

3.1.1  WG  Air  Purification  Model. 

The  Working  Group  consolidated  the  eight  high  priority  technology  categories 
into  five  technology  categories.  Table  6  shows  the  technology  categories  from  the  Master  Plan 
and  the  consolidated  categories  from  the  Working  Group  session. 


Table  6.  Air  Purifica 

tion  Model  Structures 

Master  Plan  Categories 

Working  Group  Categories 

Open/Closed  02  Supply 

Open/Closed  02  Supply 

Catalytic  Oxidation 

Catalytic  Oxidation 

Regenerable  Technologies 

Regenerable  Technologies 

Non-Carbon  Materials 

Activated/Impregnated  Carbon 

Single  Pass 

Engineered  Composite  Materials 

1 _ 

Non-Membrane  Filters 

Aerosol-Particulate  Removal 

Fiber  Filter  Treatments 

The  first  input  to  the  WG  technology  investment  model  was  an  assessment  of  the 
payoffs,  or  marginal  value,  of  increased  spending  in  the  technology.  In  other  words,  how  much 
better  would  the  expected  outcome  be  if  we  increased  investment  from  a  “low”  level  to  a 
“moderate”  level?  How  much  better  would  it  be  if  we  increased  investment  from  a  “moderate” 
level  to  a  “high”  level? 

The  AP  Working  Group  expressed  a  highly  “risk  seeking”  preference  toward 
investment  in  this  technology  development  environment.  In  other  words,  participants  believe 
that  there  is  not  much  relative  benefit  to  achieving  only  minimal  or  partial  successes  (1  to  10%). 
A  significant  increase  in  benefit  only  comes  from  a  completely  successful  development  effort 
(Table  7). 


Table  7.  Comparison  of  Relative  Benefit 


Initial  Benefit  Levels 

WG  Benefit  Levels 

Complete  Success 

100 

100 

Partial  Success 

55 

10 

Minimal  Success 

10 

1 

No  Success 

0 

0 
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For  purpose  of  comparison  to  the  initial  model  developed  by  the  BAM,  the  initial 
benefit  levels  were  used  in  the  WG  EQUITY  model.  The  revised  WG  benefit  levels  were  then 
used  to  see  what  impact  it  would  have  on  the  resulting  order  of  buy. 

The  AP  Working  Group  then  developed  relative  weights  for  the  revised 
technology  categories.  The  weights  shown  in  Table  8  are  “global  weights”  meaning  that  the 
weights  in  each  column  are  normalized  to  sum  to  1.0. 


Table  8.  Working  Group  Weights  for  AP 


Master  Plan  Categories 

Initial  Model 
Weights 

Working  Group 
Categories 

WG  Model 
Weights 

Open/Closed  02  Supply 

.21 

Open/Closed  02  Supply 

.03 

Catalytic  Oxidation 

.19 

Catalytic  Oxidation 

.11 

Regenerable  Technologies 

.17 

Regenerable  Technologies 

.40 

Non-Carbon  Materials 

.08 

Single  Pass 

.28 

Activated/Impregnated  Carbon 

.13 

Engineered  Composite  Materials 

.06 

Non-Membrane  Filters 

.08 

Aerosol-Particulate 

Removal 

.18 

Fiber  Filter  Treatments 

.08 

The  WG  model  weights  varied  significantly  from  the  initial  model.  The  WG 
discounted  the  potential  impact  of  the  02  Supply  technologies  because  participants  believed  they 
would  not  be  feasible  for  the  range  of  future  systems  requiring  collective  protection.  The  weight 
on  CatOx  was  reduced  for  similar  reasons. 

The  WG  redistributed  more  weight  to  Regenerable  Technologies  because  of  the 
great  potential  to  revolutionize  the  logistical  support  required  of  collective  protection  systems  in 
the  field. 


Finally,  the  working  group  assessed  probabilities  of  success  for  each  funding 
level.  These  are  shown  in  Table  9,  along  with  the  priorities  and  probability-adjusted  benefits. 

3.1.2  WG  AP  Technology  Investment  Strategy. 

The  WG  Air  Purification  model  structure  is  shown  in  Figure  14.  Each  technology 
category  was  treated  as  a  separate  investment  area  (row)  within  the  model.  For  each  potential 
investment  level  along  a  row,  the  cumulative  costs  and  probability-adjusted  benefits  were 
assigned  (from  Table  9). 
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Table  9.  Air  Purification  WG  Model  Inputs 


Technologies 

Program  Outcome  Probability  of 
Success 

Technology 

Weight 

Probability 

Adjusted 

Benefits 

Mnimal 

Partial 

Corrplete 

Total 

Prob. 

02  Supply 

toniestni^Le^  •  ^ 

IHBIS 

5.85' . 

gglgH 

5!jS 

Uiisi 

3% 

17.7 

0.65 

mgji 

0.05 

WISH 

^5 

HHiii 

0.45 
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Figure  14.  WG  Air  Purification  Technology  Investment  Model 
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The  model  produces  a  Pareto  diagram  of  all  possible  combinations  of  funding 
levels  with  a  maximum  of  1000  relative  benefit  points  (Figure  15).  However,  even  if  the  highest 
investment  was  made  on  all  technologies  (total  of  $66  M)  the  BAM  could  not  expect  to  achieve 
more  than  about  86%  of  the  maximum  benefit  because  of  uncertainty  about  the  success  of  the 
development  efforts  (856  out  of  1000  points). 


BENEFrrs 


WeighM  Fveforonca  Valuer 


Figure  15.  WG  AP  Pareto  Diagram 


The  green  shaded  area  at  the  top  of  the  diagram  shows  where  these  “infeasible” 
strategies  lie.  The  set  of  optimum  frontier  strategies,  therefore,  lies  along  the  top  edge  of  the 
yellow  shaded  area  of  feasible  strategies. 

This  WG  model  compares  with  the  initial  AP  model  as  shown  in  Table  10. 


Table  10.  Comparison  of  AP  Models 


Maximum  Investment 

Total  Possible  Benefit 

Initial  Model 

$64.2  M 

593 

WG  Model 

$66  M 

856 

There  was  a  small  difference  in  the  maximum  investments  because  the  working 
group  made  slightly  different  assumptions  as  they  rolled  the  eight  categories  into  five  categories. 
The  difference  in  total  possible  benefits  reflects  the  working  group’s  more  optimistic 
assessments  of  probabilities  of  successful  development  efforts. 
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As  opposed  to  the  initial  model,  where  the  frontier  curve  was  smooth  with  no 
clear  “knee,”  the  WG  model  shows  a  definite  best  value  strategy.  The  “F”  point  along  the 
frontier  shows  the  best  value  strategy,  where  for  a  little  more  than  20%  of  the  total  cost,  or 
$14.5  M,  the  program  could  achieve  593  points,  or  nearly  70%  of  the  feasible  benefit.  This  WG 
AP  best  value  strategy  is  shown  in  Figure  16. 


j  ■  Frontier  package 

■BDI 

Frontier  Pack  #4 

Preference  Values 

COSTS 

BENEFITS 

TECH 

LEVEL 

Cost 

Benefit 

Total 

Total 

1  02Supply 

1  None 

0 

0 

0 

0 

2  CatOx 

>  1  None 

0 

0 

0 

0 

3  Regen 

2  Low 

6500 

6500 

247 

247 

4  Single  Pass 

<  3  Mod 

5000 

5000 

242 

242 

;  5  Aero  Part  Rem 

2  Lov>/ 

3000 

3000 

103 

103 

Frontier  package 

14500 

14500 

593 

593 

Next  Package 

19500 

19500 

663 

663 

Previous  Pack  #3 

12000 

12000 

555 

555 

Figure  16.  AP  WG  Best  Value  Strategy 


For  each  technology  category,  the  recommended  level  of  investment  is  shown  in 
the  LEVEL  column.  The  RED  arrow  indicates  the  level  that  would  be  increased  next  if 
additional  funds  were  invested  (the  next  “buy”).  The  RED  Next  Package  #5  shows  the  total  cost 
and  benefit  of  that  “next”  strategy.  The  BLUE  arrow  indicates  the  level  that  would  be  reduced 
next  if  funds  were  cut  (the  next  “sell”).  The  BLUE  Previous  Package  #3  shows  the  total  cost  and 
benefit  of  that  “previous”  strategy. 

The  COSTS  and  BENEFITS  columns  show  the  costs  and  benefits  for  each 
technology  category  at  the  recommended  levels.  The  Total  columns  are  the  same  as  the  Cost  and 
Benefit  columns  because  there  are  only  one  type  of  cost  and  one  type  of  benefit  in  the  model. 
Using  the  “risk  seeking”  benefit  values  that  the  AP  working  group  assessed,  instead  of  the 
BAM’s  “risk  neutral”  values  (Table  7),  gives  model  results  that  are  very  similar  in  the  best  value 
investment  strategy  (Figure  17).  The  difference  lies,  as  expected,  in  a  reduction  in  the  overall 
assessed  value  of  the  strategy  (477  benefit  points  versus  593  benefit  points  using  “risk  neutral” 
benefit  values).  This  reflects  a  view  that  low  to  moderate  investments  in  the  AP  technology 
thrust  area  would  not  return  proportional  benefits — it  would  be  more  efficient  to  invest  at  high 
levels  or  not  at  all. 
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1  ■  Frontier  package 

1 
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Cost 
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Benefit 

Total 

1  02Supply 

1  None 

0 

0 

0 

0 

2  CatOx 

)  None 

0 

0 

0 

0 

3  Regen 

2  Lo-a-’ 

6500 

6500 

191 

191 

4  Single  Pass 

<  3  Mod 

5000 

5000 

216 

216 

5  Aero  Part  Rem 

2  Low 

3000 

3000 

70 

70 

^  Frontier  package 

14500 

14500 
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477 

Next  Package  #5 

17000 

17000 

515 

515 

Previous  Pack  #3 

12000 

12000 

424 

424 

Figure  17.  AP  WG  Best  Value  Strategy  with  Risk  Seeking  Benefit  Values 


This  WG  best  value  strategy  is  significantly  different  from  the  BAM’s  initial  best 
value  strategy  (Table  1 1).  Even  investing  50%  of  the  approximately  $64  M  maximum,  the 
BAM’s  initial  assessment  model  showed  that  only  a  relatively  modest  64%  of  the  feasible  benefit 
could  be  achieved.  In  contrast,  the  WG  believed  that  only  a  20%  investment  could  achieve  high 
benefits  of  70%.  If  the  investment  level  in  the  BAM’s  initial  model  is  reduced  to  20%,  the 
optimal  strategy  only  achieves  38%  of  expected  benefits. 


Table  1 1.  Comparison  of  Air  Purification  Best  Value  Strategies 


Percent  of  Total 
Possible  Investment 

Percent  of  Total 
Possible  Benefits 

Initial  Model 

50% 

64% 

WG  Model 

20% 

70% 

Initial  Model  w/reduced  investment 

20% 

38% 

The  complete  Order  of  Buy  for  the  WG  AP  is  shown  in  Figure  18.  According  to 
the  WG,  this  is  the  investment  list  that  the  BAM  should  follow  in  order  to  maximize  the  value  of 
his  investments  at  any  cumulative  level  of  funding.  The  best  value  strategy  is  shown  at 
increment  #4. 


Because  the  WG  consolidated  the  technology  categories  from  8  to  5,  it  is  not 
possible  to  directly  compare  the  differences  among  the  Order  of  Buy  lists.  However,  comparing 
Figure  18  to  Figure  5,  it  is  easy  to  see  that  the  biggest  difference  is  the  almost  total  lack  of  value 
placed  on  the  02  Supply  technology  by  the  WG.  Because  02  Supply  is  an  expensive  program 
(Table  9),  the  WG  was  able  to  show  more  relative  benefit  from  an  investment  strategy  focusing 
on  Single  Pass  and  other  technologies. 
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■  Order  of  Buy 

IBD 
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m 
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TECH 

1  -  02Supply 
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INC  CUM 

0  0 

0  0 
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3 
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2- 
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51000 
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#13 
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5 
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1  - 
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4 
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10000 

66000 

7 

856 

Figure  18.  AP  WG  Order  of  Buy 


3.2  Shelters. 

The  Shelters  Working  Group  was  unable  to  validate  the  Shelter  Materials  and 
Treatments  initial  model  because  the  participants  believed  that  the  model  was  too  detailed  for  the 
level  of  information  available  at  this  time.  As  an  alternative,  the  group  developed  a  new  model, 
which  included  the  Shelter  Materials  and  Treatments  technologies  and  the  CP  Critical 
Components  technology  thrust  area. 

3.2.1  WG  Shelters  Model. 

The  Working  Group  combined  the  seven  high  potential  Materials  and  Treatments 
technologies  into  a  single  rating  area  called  Materials  (Table  12),  which  included  the  technology 
categories  of  Moisture- Vapor  Permeable  Materials,  Impermeable  Barrier  Materials,  and  Material 
Treatments. 
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Table  12.  Consolidated  Materials  Technologies 


Master  Plan  Categories 

Working  Group  Category 

1 .  Selectively  Permeable  Membranes 

1.  Materials 

2.  Semipermeable  Membranes 

3.  Homogeneous  Materials 

4.  Heterogeneous  Materials 

5.  Multilayer  Materials 

6.  Active/Reactive  Treatments 

7.  High  Surface  Tension  Treatments 

The  Critical  Components,  Structural  Supports  and  Studies  and  Analyses 
technology  categories  identified  by  the  Fall  2001  Battelle  meeting  panel  were  consolidated  from 
eight  categories  to  five  categories  (Table  13). 


Table  13.  Consolidated  Critical  Component  Technologies 


Battelle  Meeting  Technologies 

Working  Group  Technologies 

1 .  Novel  Closures,  Seams  and  Seals 

1 .  Closures  and  Seals 

2.  Seams 

2.  Air  Beam  Technologies 

3.  Structural  Supports 

3.  Tension  Frame/Fabric 

4.  Composite  Frames 

5.  Next  Gen  Air  Locks,  Barriers,  Doors 

4.  Airlocks 

6.  Integrated  Power  and  ECU  Control  System 

5.  Environmental  Controls 

7.  Threat  Mitigation  Methodologies 

**Not  Included  in  WG  model 

8.  Energy  Efficient  Materials  Development 

**Not  Included  in  WG  model 

In  the  Shelters  Area,  the  working  group  had  difficulty  initially  assessing  the 
probabilities  of  success  in  terms  of  minimal,  partial,  and  complete  success,  because  the  transition 
objectives  were  not  well  defined.  Instead,  for  each  technology  area,  the  working  group  assessed 
the  probability  of  achieving  some  kind  of  transition  (incremental  or  breakthrough)  every  two-to- 
three  years  over  the  ten-year  program  period  given  a  level  of  funding.  Incremental  transitions 
allow  marginal  improvements  to  the  existing  systems.  Breakthrough  transitions  allow  much 
better  systems  to  be  developed.  This  “Success”  probability  was  entered  in  the  “Complete” 
success  column  (Table  14). 

The  WG  used  the  assumption  that  a  Moderate  funding  level  was  “nominal”  to 
pursue  6.2  development  in  this  area,  and  that  a  Low  and  High  funding  levels  cost  50%  less  and 
50%  more  respectively. 
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Table  14.  WG  Shelters  Model  Inputs 


Technologies 

Program  Outcome  Probability  of 
Success 

Technology 

Weight 

Probability 

Adjusted 

Benefits 

Minima! 

Partial 

Complete 

Total 

Prob. 

HHEZ! 

Buaiiin 

0.00 

0.50 

IKIiliB 

50.0 

Materials 

;  Investment  Level  ($000) 

■IS5E1HE 

0.00 

0.75 

■KiTil 

50% 

75.0 

fTDD 

Biisl 

tr55 

HSfil 

5515 

Technology  Benefit 

0 

0 

100 

0.00 

0.60 

■mM 

60.0 

Airlocks 

Investmehl  Level  ($000)  . 

0.00 

0.90 

■Mil 

16% 

90.0 

CUD 

0.96 

Hilil 

5515 

Technology  Benefit 

0 

0 

100 

•  tow 

0.00 

0.40 

0.60 

■WtTil 

60.0 

.  Investment  Cev^!  ($000), 

wmm 

0.00 

0.80 

■mill 

6% 

80.0 

Environ.  Control 

mamim 

■lid 

■ESlI 

5015 

Technology  Benefit 

0 

0 

100 

tow 

0.00 

0.20 

■iiuM 

20.0 

Investment  level  ($000)  . 

Kssim 

0.00 

0.40 

MKilil 

2% 

40.0 

Structural  Support 

KfiSstSil 

mmtm 

on 

HSlSl 

5015 

Technology  Benefit 

0 

0 

100 

0.00 

0.60 

■MM 

60.0 

Closures/Seais 

Investment  Levd  ($000) 

0.00 

0.10 

0.90 

■miiM 

15% 

■SSSiSHi 

TW 

tm 

100.0 

Technology  Benefit 

0 

0 

100 

0.00 

0.35 

0.65 

65.0 

Seams 

Investment  Level  ($000) 

mmms, 

0.00 

MtPSM 

0.80 

MBTtl 

11% 

80.0 

mamm 

■sm 

H2I 

5T0 

Technology  Benefit 

0 

0 

100 

The  working  group  assessed  the  Materials  technology  thrust  area  as  a  roll-up  of 
the  three  technology  categories:  Moisture-Vapor  Permeable  Materials,  Impermeable  Barrier 
Materials  and  Material  Treatments.  As  stated  above,  the  working  group  assumed  that  the 
technical  approach  in  this  area  would  be  to  make  multiple,  incremental  improvements  or 
breakthrough  transitions  every  few  years  over  the  ten-year  timeframe. 

The  Shelters  group  assessed  the  probability  of  “success”  -  “success”  was  only 
considered  at  the  “Complete  Success”  level  by  the  WG  -  for  each  investment  level,  but  did  not 
attempt  to  assign  probabilities  to  various  degrees  of  success  (Complete,  Partial,  and  Minimal). 
The  group  made  separate  success  probability  assessments  for  Impermeable  Materials  and 
Permeable  Materials,  but  did  not  assess  success  probabilities  for  Material  Treatments.  The  group 
then  assessed  an  “average”  success  probability  across  all  Material  technologies  (Table  15). 


Table  15.  1 

Materials 

Approx. 
Funding  for  10 
Years 

P(Success) 

Impermeable 

Materials 

P(Success) 

Permeable 

Materials 

P(Success) 

Average 

High  Funding  Level 

$39M 

98% 

60% 

85% 

Moderate  Funding  Level 

$26M 

95% 

50% 

75% 

Low  Funding  Level 

$13M 

85% 

10% 

50% 
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The  moderate  level  of  funding  in  Table  15  was  obtained  by  summing  the  ’’high” 
level  of  funding  as  assessed  by  the  BAM.  The  WG  then  added  50%  and  subtracted  50%  to  get 
the  “low”  and  “high”  levels  for  the  above  chart. 

The  working  group  developed  the  following  representative  transition  objectives 
for  the  Airlocks  technology  category  based  on  a  Moderate  funding  level. 

•  Reduce  dwell  time  (<  3  min) 

•  Reduce  weight  and  volume 

•  Reduce  purge  air  volume  (total  amount  of  air) 

•  Reduce  energy  loss  due  to  air  loss 

The  group  then  determined  a  funding  level  for  a  ten-year  period  along  with  a 
probability  of  success  at  each  funding  level  for  the  Airlock  technologies  (Table  16). 


Table  16.  Airlocks 


WG  Model 
Funding 
for  10  Years 

Initial  Model 
Funding 
for  10  Years 

WG 

Model 

P(Success) 

Initial 

Model 

P(Success) 

High  Funding  Level 

$6.75M 

$6M 

95% 

90% 

Moderate  Funding  Level 

$4.5M 

$3M 

90% 

70% 

Low  Funding  Level 

$2.25M 

$1M 

60% 

55% 

The  working  group  developed  the  following  representative  transition  objectives 
for  the  Closures/Seals  technology  category  based  on  a  Moderate  funding  level. 

•  Reduce  leakage  rate 

•  Improve  ease  of  manufacturing 

•  Improve  ease  of  use 

•  Contractors  to  make  incremental  improvements 

•  Transitioning  something  to  6.4,  incremental  or  breakthrough,  within  3  years 

The  group  then  determined  a  funding  level  for  a  ten-year  period  along  with  a 
probability  of  success  at  each  funding  level  for  the  Closures  and  Seals  technologies  (Table  17). 


Table  17.  Cl 

osures  and  Seals 

WG  Model 
Funding 
for  10  Years 

Initial  Model 
Funding 
for  10  Years 

WG 

Model 

P(Success) 

Initial 

Model 

P(Success) 

High  Funding  Level 

$8.25M 

$11M 

100% 

90% 

Moderate  Funding  Level 

$5.5M 

$6M 

90% 

70% 

Low  Funding  Level 

$2.7M 

$3M 

60% 

50% 
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The  working  group  developed  the  following  representative  transition  objectives 
for  the  Environmental  Controls  technology  category  based  on  a  Moderate  funding  level. 

•  Weight  and  Volume 

•  Energy  Demands 

•  Maintainability 

•  Integrated  System  (Filter,  Blower,  Heat,  AC,  Power) 

•  Scalability 

•  Contractors  to  make  incremental  improvements 

The  group  then  determined  a  funding  level  for  a  ten-year  period  along  with  a 
probability  of  success  at  each  funding  level  for  the  Environmental  Controls  technologies 
(Table  18). 


Table  18.  Environmental  Controls 


WG  Model 
Funding 
for  10  Years 

Initial  Model 
Funding 
for  10  Years 

WG 

Model 

P(Success) 

Initial 

Model 

P(Success) 

High  Funding  Level 

$6.75M 

$6M 

90% 

80% 

Moderate  Funding  Level 

$4.5M 

$3M 

■IhHI 

Low  Funding  Level 

$2.25M 

$1M 

45% 

The  working  group  developed  the  following  representative  transition  objectives 
for  the  Seams  technology  category  based  on  a  Moderate  funding  level. 

•  Manufacturability 

•  Durability 

•  Efficacy 

•  Universally  Applicable 

•  Field  Repairable 

•  Multiple  Techs 

The  group  then  determined  a  funding  level  for  a  ten-year  period  along  with  a 
probability  of  success  at  each  funding  level  for  the  Seams  technologies  (Table  19). 


Table 

19.  Seams 

WG  Model 

Initial  Model 

WG 

Initial 

Funding 

Funding 

Model 

Model 

for  10  Years 

for  10  Years 

P(Success) 

P(Success) 

High  Funding  Level 

$8.25M 

$11M 

100% 

90% 

Moderate  Funding  Level 

$5.5M 

$6M 

100% 

70% 

Low  Funding  Level 

$2.75M 

$3M 

100% 

50% 
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The  working  group  developed  the  following  representative  transition  objectives 
for  the  Structural  Support  technology  category  based  on  a  Moderate  funding  level. 

•  Airbeams 

•  Shelter  that  does  not  require  a  liner 

•  Integration 

•  Reduce  weight,  volume,  and  O&M 

•  Tension  Frame  Fabric 

•  Turn  into  a  CB  barrier 

•  Composite  Frame  Hinge 

The  group  then  determined  a  funding  level  for  a  ten-year  period  along  with  a 
probability  of  success  at  each  funding  level  for  the  Structural  Supports  technologies  (Table  20). 


Table  20.  Structural  Supports 


WG  Model 
Funding 
for  10  Years 

Initial  Model 
Funding 
for  10  Years 

WG 

Model 

P(Success) 

Initial  Model 
Average 
P(Success) 

High  Funding  Level 

$4.5M 

$9M 

80% 

78% 

Moderate  Funding  Level 

$3M 

$5.5M 

40% 

67% 

Low  Funding  Level 

$1.5M 

$2.5M 

20% 

55% 

The  Shelters  Working  Group  then  developed  relative  weights  for  the  revised 
technology  categories.  The  weights  shown  in  Table  21  are  “global  weights”  meaning  that  the 
weights  in  each  column  are  normalized  to  sum  to  1 .0.  The  weights  in  the  left  column  were 
developed  during  the  Master  Planning  meeting  for  the  CP  business  area. 


Table  21.  WG  Weights  for  Shelters 


Battelle  Meeting  Technologies 

Weights 

Working  Group  Technologies 

Weights 

Materials 

.50 

Materials 

.50 

1 .  Novel  Closures,  Seams  and  Seals 

.13 

1.  Closures  and  Seals 

.15 

2.  Seams 

.11 

2.  Air  Beam  Technologies 

.07 

3.  Structural  Supports 

.02 

3.  Tension  Frame/Fabric 

.01 

4.  Composite  Frames 

.03 

5.  Next  Gen  Air  Locks,  Barriers,  Doors 

.10 

4.  Airlocks 

.16 

6.  Integrated  Power  and  ECU  Control 
System 

.08 

5.  Environmental  Controls 

.06 

7.  Threat  Mitigation  Methodologies 

.05 

**Not  Included  in  Model 

8.  Energy  Efficient  Materials 
Development 

.03 

**Not  Included  in  Model 
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3.2.2 


WG  Shelters  Technology  Investment  Strategy. 


The  next  figure  shows  the  strategy  model  framework  as  modified  by  the  Shelters 
Working  Group  (Figure  19). 


1  [  j]  EQUITY  for  Windows 

-  [  c:Vnydocu~1\other\equity\cpmast~1Vcpshelte.eqw] 
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|Maximum 

1  Air  Locks 
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iMaximum 

1 

f 
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1  Environ.  Control 
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. . . . . . 
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1  Structural  Support 
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1  Closures/Seals 

|; 

None 
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Mod 
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1  Maximum 
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None 
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Mod 
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|Maximum 

t 

Figure  19.  WG  Shelters  Technology  Investment  Model 


The  model  produces  a  Pareto  diagram  of  all  possible  combinations  of  funding 
levels  with  a  maximum  of  1000  relative  benefit  points  (Figure  20).  However,  even  if  the  highest 
investment  was  made  on  all  technologies  (total  of  $73.5  M)  the  BAM  could  not  expect  to 
achieve  more  than  about  90%  of  the  maximum  benefit  because  of  uncertainty  about  the  success 
of  the  develop  efforts  (900  out  of  1000  points). 
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Figure  20.  WG  Shelters  Pareto  Diagram 
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The  green  shaded  area  at  the  top  of  the  diagram  shows  where  these  “infeasible” 
strategies  lie.  The  set  of  optimum  frontier  strategies,  therefore,  lies  along  the  top  edge  of  the 
yellow  shaded  area  of  feasible  strategies. 

It  is  not  possible  to  compare  this  WG  model  with  the  initial  Shelter  Materials  and 
Treatments  model  and  the  Critical  Components  model  because  the  model  structures  are  so 
different. 


The  WG  Shelters  model  shows  a  frontier  curve  that  is  smooth  with  no  clear 
“knee”  for  a  best  value  strategy.  However,  the  “F”  point  along  the  frontier  shows  a  suggested 
best  value  strategy,  where  for  a  little  more  than  one-third  of  the  total  cost,  or  $28  M,  the  program 
could  achieve  637  points,  or  a  little  more  than  70%  of  the  feasible  benefit.  This  WG  Shelters 
best  value  strategy  is  shown  in  Figure  21. 


Frontier  package 

IBD 

1  Frontier  Pack  W 

i 

Preference  Values 

COSTS 

BENEFITS 

TECH 

LEVEL 

Cost 

Benefit 

Total 

Total 

1 

Materials 

>  2  Low 

13000 

13000 

250 

250 

2 

Air  Locks 

3  Mod 

4500 

4500 

144 

144 

3 

Environ.  Control 

<  2  Low 

2250 

2250 

36 

36 

4 

Structural  Support 

1  None 

0 

0 

0 

0 

5 

Closures/Seals 

3  Mod 

5500 

5500 

135 

135 

6 

Seams 

2  Low 

2750 

2750 

72 

72 

Frontier  f3ackage 

28000 

28000 

637 

637 

Next  Package 

41000 

41000 

762 

762 

Previous  Pack  ^ 

25750 

25750 

601 

601 

Figure  21.  WG  Shelters  Best  Value  Strategy 


For  each  technology  category,  the  recommended  level  of  investment  is  shown  in 
the  LEVEL  column.  The  RED  arrow  indicates  the  level  that  would  be  increased  next  if 
additional  funds  were  invested  (the  next  “buy”).  The  RED  Next  Package  #8  shows  the  total  cost 
and  benefit  of  that  “next”  strategy.  The  BLUE  arrow  indicates  the  level  that  would  be  reduced 
next  if  funds  were  cut  (the  next  “sell”).  The  BLUE  Previous  Package  #6  shows  the  total  cost  and 
benefit  of  that  “previous”  strategy. 

The  COSTS  and  BENEFITS  columns  show  the  costs  and  benefits  for  each 
technology  category  at  the  recommended  levels.  The  Total  columns  are  the  same  as  the  Cost  and 
Benefit  columns  because  there  are  only  one  type  of  cost  and  one  type  of  benefit  in  the  model. 
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The  complete  Order  of  Buy  for  the  WG  Shelters  model  is  shown  in  Figure  22. 
According  to  the  WG,  this  is  the  investment  list  that  the  BAM  should  follow  in  order  to 
maximize  the  value  of  his  investments  at  any  cumulative  level  of  funding.  The  best  value 
strategy  is  shown  at  increment  #7. 


■I  Order  of  Buy 


p 


la 


Order  of  Buy 


COSTS  BENEFITS 


TECH 

LEVEL 

INC 

CUM 

INC 

CUM 

#0 

1  - 

Materials 

1 

None 

0 

0 

0 

0 

#0 

2- 

Air  Locks 

1 

None 

0 

0 

0 

0 

#0 

3- 

Environ.  Control 

1 

None 
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0 

0 

0 
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4- 

Structural  Support 
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None 

0 

0 

0 

0 

m 

5- 

Closures/Seals 

1 

None 
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0 

0 

0 

m 

6- 

Seams 

1 

None 

0 

0 

0 

0 

#1 

2  - 

Air  Locks 

2 

Low 

2250 

2250 

96 

96 

#2 

5- 

Ciosures/Seals 

2 

Low 

2750 

5000 

90 

186 

6  - 

Seams 

2 

Low 

2750 

7750 

72 

258 

#4 

2- 

Air  Locks 

3 

Mod 

2250 

10000 

48 

306 

#5 

1  - 

Materials 

2 
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13000 

23000 
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556 

m 

5- 
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3 

Mod 

2750 

25750 

45 
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w 

3- 

Environ.  Control 
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2250 

28000 

36 
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HQ 

1  - 
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3 

Mod 

13000 

41000 

125 
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6- 

Seams 

3 

Mod 

2750 

43750 

17 

778 

#10 

6- 

Seams 

4 

High 

2750 

46500 

15 

793 

#11 

5- 

Closures/Seals 

4 

High 

2750 

49250 

15 

808 

#12 

3- 

Environ.  Control 

3 

Mod 

2250 

51500 

12 

820 

#13 

1  - 

Materials 

4 

High 

13000 

64500 

50 

870 

#14 

4- 

Structural  Support 

4 

High 

4500 

69000 

16 

886 

*15 

2- 

Air  Locks 

4 

High 

2250 

71250 

8 

894 

#16 

3- 

Environ.  Control 

4 

High 

2250 

73500 

6 

900 

Figure  22.  WG  Shelters  Order  of  Buy 


Finally,  the  Shelters  Working  Group  conducted  a  direct  resource  allocation 
exercise  to  assess  the  participants’  “instinctive”  judgment  about  how  available  near  term  funds 
should  be  spent.  The  participants  were  asked,  “If  $1  million  were  available  in  the  next  fiscal 
year  for  the  Shelters  Area,  what  percent  should  be  allocated  to  each  of  the  six  technology 
categories?”  The  six  individual  participants’  assessments  (shown  as  A  through  F)  and  the 
numerical  average  are  shown  in  Table  22. 
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Table  22.  WG  Shelters  Direct  Resource  Allocation 


Shelters 
Materials 
Airlocks 
ClosuresSeals 
Env  Control 
Seams 
Struct  Spt 

Total  Percent 


58.5 

25 

25 

30 

20 

30 

12 

20 

25 

20 

30 

25 

15 

15 

20 

20 

20 

10 

10 

20 

20 

10 

10 

15 

4.5 

15 

5 

20 

10 

15 

0 

5 

5 

0 

10 

5 

100 


100 


100 


100 


100 


100 


Average 

31.4 

22.0 

16.7 

14.2 

11.6 

4.2 

100.0 


The  equivalent  $1  M/year  strategy  in  the  WG  Shelters  model  is  shown  in 
Figure  23,  which  has  a  frontier  package  at  exactly  $  lOM  over  ten  years.  Because  of  the  large 
increments  of  funding  used  to  build  the  model,  the  entire  $1M  would  be  allocated  to  only  three 
technology  categories:  Air  Locks  (45%),  Closures/Seals  (27.5%),  and  Seams  (27.5%). 


Hfll 

im 

■ll 

.ski 
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Preference  Values 

COSTS 

BENEFITS 

TECH 

LEVEL 

Cost 

Benefit 

Total 

Total 

1 

Materials 

>  1  None 

0 

0 

0 

0 

2 

Air  Locks 

<  3  Mod 

4500 

4500 

144 

144 

3 

Environ.  Control 

1  None 

0 

0 

0 

0 

4 

Structural  Support 

1  None 

0 

0 

0 

0 

5 

Closures/Seals 

2  Low 

2750 

2750 

90 

90 

^6 

Seams 

2  Low 

2750 

2750 

72 

72 

Frontier  package 

10000 

10000 

306 

306 

Next  Package  ^ 

23000 

23000 

556 

556 

Previous  Pack 

7750 

7750 

258 

258 

Figure  23.  WG  Shelters  Strategy  for  $1M  per  Year 


However,  the  “next  package”  priority  increment  to  the  model  is  the  Level  2  (Low 
Investment)  in  Materials  ($1 .3M  per  year).  At  this  level  of  total  funding  ($2.3M  per  year),  the 
model  shows  results  very  similar  to  the  “instinctive”  allocation  (Table  23). 


Table  23.  Comparison  of  Resource  Allocation  Results  for  Shelters 


Technology  Category 

Direct  $1M  Allocation 

Model  $2.3  M  Allocation 

Materials 

31.4% 

56.5% 

Air  Locks 

22% 

20% 

Closures/Seals 

16.7% 

12% 

Env  Control 

14.2% 

0% 

Seams 

11.6% 

12% 

Structural  Spts 

4.2% 

0% 
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4. 


CONCLUSIONS 


The  Working  Group  for  the  Air  Purification  Technology  Thrust  Area  was  not  able 
to  confirm  the  optimal  investment  strategy  for  the  AP  area  as  identified  by  the  BAM.  As  shown 
in  Table  24,  the  strategies  are  very  different.  The  differences  lie  in  terms  of  the  gross  amount  of 
investment  needed  to  gain  the  bulk  of  the  potential  benefits,  and  in  the  assessment  of  the 
potential  value  of  02  Supply  and  Catalytic  Oxidation  technologies. 


Table  24.  Comparison  of  AP  Initial  and  WG  Best  Value  Strategies 


Initial  Categories 

10  year 

Investment  ($M) 

WG  Categories 

10  year 

Investment  ($M) 

Open/Closed  02  Supply 

$4.5 

Open/Closed  02  Supply 

$0 

Catalytic  Oxidation 

$5.0 

Catalytic  Oxidation 

$0 

Regenerable  Technologies 

$3.85 

Regenerable  Technologies 

$6.5 

Non-Carbon  Materials 

$1.75 

Single  Pass 

$5.0 

Activated/Impregnated 

Carbon 

$4.5 

Engineered  Composite 
Materials 

$1.8 

Non-Membrane  Filters 

$4.5 

Aerosol-Particulate  Removal 

$3.0 

Fiber  Filter  Treatments 

$3.0 

Total 

$28.9 

Total 

$14.5 

The  Working  Group  for  the  Shelters  and  Critical  Components  Technology  Thrust 
Areas  developed  a  resource  allocation  model  at  a  high  level  of  aggregation,  combining  all  the 
Shelter  Materials  and  Treatments  into  a  single  “investment  area.”  It  is  therefore  not  possible  to 
show  whether  the  group  confirmed  the  BAM’s  initial  investment  strategy  for  shelter  materials. 
However,  for  Critical  Components,  the  working  group  came  closer  to  confirming  the  optimal 
investment  strategy  as  identified  by  the  BAM  (Table  25). 

As  a  next  step,  the  COLPRO  community  should  seek  to  standardize  the 
investment  categories  it  will  use  to  allocate  resources  in  each  budget  cycle. 

In  addition,  the  BAM  should  convene  SME  panels  each  year  to  reassess  and 
update  the  Master  Plan  resource  allocation  model  to  insure  a  steady  course  in  investment 
implementation  until  the  next  FEA  and  Master  Planning  cycle. 

Finally,  the  technology  thrust  areas  should  be  examined  together  to  produce  a 
single,  integrated  investment  strategy  for  COLPRO.  This  will  insure  the  optimal  allocation  of 
resources  across  the  business  area. 
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Table  25.  Comparison  of  Shelters  Initial  and  WG  Best  Value  Strategies 


Initial  Categories 

10-Year 

Investment  ($M) 

WG  Categories 

10-Year 

Investment  ($M) 

Materials 

$5 

$13 

1.  Novel  Closures,  Seams 
and  Seals 

$3 

. 

1 .  Closures  and  Seals 

$5.5 

2.  Seams 

$2.75 

2.  Air  Beam  Technologies 

$1 

3.  Structural  Supports 

$0 

3.  Tension  Frame/Fabric 

$0.5 

4.  Composite  Frames 

$1 

5.  Next  Gen  Air  Locks, 
Barriers,  Doors 

$1 

4.  Airlocks 

$4.5 

6.  Integrated  Power  and 

ECU  Control  System 

$1 

5.  Environmental  Controls 

$2.25 

7.  Threat  Mitigation 
Methodologies 

$1 

**Not  Included  in  Model 

8.  Energy  Efficient 

Materials  Development 

0 

**Not  Included  in  Model 

Total 

$13.5 

Total 

$28 
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SUMMARY  TABLES  FOR  MASTER  PLAN 
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Table  A-1.  Air  Purification  High  Potential  Technologies  Assessment  (Continued) 
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Table  A-2.  Shelters  High  Potential  Technologies  Assessment 
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Table  A-2.  Shelters  High  Potential  Technologies  Assessment  (Continued) 
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APPENDIX  B 


WORKING  GROUP  PARTICIPANTS 


ORGANIZATION 

Working 

Group 

NAME 

COMM  # 

E-Mail 

COLPRO  BAM 

AFRL/MLQ  (USAF) 

AP& 

Shelters 

Mr.  Bruce  Nielsen 

(850) 

283-6227 

Bruce. nielsen(5)tvn(jall.af.mil 

SBCCOM/ECBC 

(USA) 

Filtration 

AP 

Pri:  Mr.  Chris 
Karwacki 

Alt:  Mr.  Jeff  Haney 

(410) 

436-5704 

ChristoDher.karwacki(Q)sbccom.aDae 

a.armv.mi! 

JTCOPS  PM  (USA) 

AP& 

Shelters 

Mr.  Jorge  Christian 

(410) 

436-5512 

Jorqe.christian(®sbccom.aDaea.arnnv 

.mil 

AFRUMLQF  (USAF) 

Filtration 

AP& 

Shelters 

Dr.  Joe  Wander 

(850) 

283-6240 

Joe.wander(S)tvndalLaf.mil 

HSW/YACN  (USAF) 

CB  Program  Office 
Battelle 

AP& 

Shelters 

Mr.  Abe  Saenz 

(210) 

536-3434 

Abe. saenz(S!  brooks.af.mil 

AAC/WMO  (USAF) 
Combat  Support 
Systems,  Program 
Office 

CPE  integration 
Sverdrup 

AP& 

Shelters 

Mr.  Scott  Matheson 

(850) 

882-4684 
Ext.  337 

Scott.matheson(0)ealin.af.mil 

USA  NATICK 

Shelter  Materials 

Shelters 

Mr.  Kristian  Donahue 

(508) 

233-5202 

Donahue. Kristian(a)natick.armv.mil 

University  Contacts: 

Clemson  University 
(Materials) 

Shelters 

Dr.  Christine  Jarvis 

(864) 

646-8454 

C  wi  rv(®  clemson .  ed  u 

Decision  Advantage 

AP& 

Shelters 

Mr.  Freeman  Marvin 

(703) 

593-5335 

ffmarvin(a),decision-advantaae.com 

SBCCOM/ECBC 

(USA) 

Decision  Analysis 
Team 

AP& 

Shelters 

Mrs.  Genna  Buckless 

(410) 

436-9788 

Genna. Buck!ess(a)sbccom.aDaea.ar 
mv.mil 

SBCCOM/ECBC 

(USA) 

Decision  Analysis 
Team 

AP& 

Shelters 

Ms.  Trish  Vargo 

(410) 

436-4775 

Trsiha.Varao(5)sbccom.aDaea.armv. 

mil 

SBCCOM/ECBC 

(USA) 

Decision  Analysis 
Team 

AP& 

Shelters 

Mr.  John  Walther 

(410) 

436-3569 

John.Walther(a>sbccom.aDaea.armv. 

mil 
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APPENDIX  C 


PRESENTATION  TO  THE  71®^  MORS  SYMPOSIUM 

Slide  1 


Presentation  to  the  71st  MORS  Symposium 
Working  Group  28  -  Decision  Analysis 


Technology  Investment  Strategy  Annex: 
Collective  Protection  Front  End  Analysis  and 
Master  Plan  Report 

Genna  Lee  Buckless 
Freeman  Marvin 
Trish  Vargo 
John  Walther 

Decision  Analysis  Team/ECBC 


12  June  2003 
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Outline 


•  Background 

•  Purpose  &  Impact  of  the  Analysis 

•  Analysis  Approach 

—  Tools,  objectives,  analysis  framework,  results 

•  Lessons  Learned 
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Background 


Collective  Protection  Front  End  Analysis 
~  Ranking  of  viable  technologies  relative  to  application  areas 
Collective  Protection  Master  Plan 

-  Select  technologies  evaluated  against  other  considerations 

-  2  products 

•  ID  Techs  for  the  Tech  Base  Program 

•  Develop  a  strategic  resource  allocation  model 

-  4  steps  in  process 

•  Define  CP  BA  framework 

•  Assess  high  potential  technologies 

•  Prioritize  techs  and  establish  time  frames  for  transition 

•  ^Develop  planning  models  &  examine  alternative  program 
strategies 
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Purpose  &  Impact 
of  Analysis 


•  Purpose 

-  Develop  and  examine  alternative  funding 
strategies 

-  Use  funding  strategies  to  create  an  investment 
portfolio  which  is  optimized  over  10  years 

•  Impact 

-  Development  of  method  to  make  strategic 
funding  decisions  for  R«feD  programs 
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Analysis  Approach 


a)  Tools 

b)  Objectives 

c)  Analysis  Framework 

d)  Results 
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Tools 
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eoOEWOOD 


Objectives 


1)  Initial  Analysis 

•  Examine  alternative  funding  strategies 

•  Determine  optimal  set 

2)  Workgroup  Analysis 

•  Validate  initial  models  developed  by 
BAM 
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Initial  Analysis 
Framework  Steps 


EDGEWOOD 


•Create  decision  tree  model 
^•Create  strategy  tables 
•Estimate: 

•Cost  of  alternative  investment  levels 
•Probability  of  success 
•Technology  benefit  levels 
•Assess  overall  technology  weights 
,rCompile  data 

•Compute  probability  adjusted  benefit  levels 

r^Examine  all  possible  combinations  of  technology  investments 

•Create  an  investment  portfolio 
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Initial  Analysis  ®  Miss; 
Framework 

Decision  Tree  and  Influence  Diagram  showing  Strategy  Table 


EDGEWOOD 


!SLin_Aclive_Re3ctive_  Low  Invest 


ED 
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Initial  Analysis  ^  yiissiiisgs 
Framework 


Strategy  Table 


rvci 

hvesl  Non- 

OSsupptv’ 

Regen? 

Carton’ 

Memb 

F«efS’ 

1  Reduced 

No  tivest 

No  Invest 

No  Invest 

No  Invest 

{f]sti%tusquo 

Low  hivesl 

■OD 

Low  Invest 
□ 

Low  Invest 

□aa 

|e4lar.ced 

Mod  Invest 

□  Agwp^'lve 

Mod  In-<e5t 

mm 

mmm 

[~~}  Increased 

Hctitnvcsl 

□  □□ 

High  Invest 

mmam 

Hgti  Invest 

mm 

bfah  Invest 

□□a 

^Brucel 

QBiuce2 

Qm  Investment 
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Pareto  Diagram  Frontier  Package 
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Initial  Results 

Order  of  Buy 


l|  ■  Order  of  Buy 

■■■1 

IB 

■  -Mx| 

Order  of  Buy 

COSTS 

BENEFITS  \ 

TECH 

LEVEL 

INC 

CUM 

INC 

CUM 

m 

1 

or-’SiJi'piv 

1 

None 

0 

0 

0 

0 

m 

2- 

CafO;; 

1 

None 

0 

0 

0 

0 

m 

3  - 

Renf^n 

1 

None 

0 

0 

0 

0 

m 

4  - 

1 

None 

0 

0 

0 

0 

m 

5 

Not'!  Morn  Fill 

1 

None 

0 

0 

0 

0 

m 

6  - 

EngC  nivip 

1 

None 

0 

0 

0 

0 

7  ■ 

FiliFilTroa! 

1 

None 

0 

0 

0 

0 

m 

f: 

/VlC.iihnn 

1 

None 

0 

0 

0 

0 

#1 

8- 

AciCaibAfi 

2 

Low 

1500 

1500 

51 

51 

7 

FiliFtlTu?M 

2 

Low 

1500 

3000 

24 

75 

#3 

5 

Monf'^f-nriF  i!t 

3 

Mod 

3000 

6000 

48 

123 

#4 

8 

ActCafl'Cm 

3 

Mod 

1500 

7500 

20 

143 

#5 

7  - 

FiFFilTinA 

3 

Mod 

1500 

9000 

20 

163 

2  . 

CatOi 

2 

Low 

5000 

14000 

62 

225 

m 

4  ■ 

NoriC  .<tL’0n 

2 

Low 

1750 

15750 

21 

345 

m 

8  ■ 

Ac  tC  a  til  nr. 

4 

High 

1500 

17250 

18 

263 

m 

6  - 

Eiif)Cnrrip 

2 

Low 

1800 

19050 

20 

283 

#10 

3  ■ 

Rt?rjr'ri 

2 

Low 

3850 

22900 

40 

322 

#11 

1  ■ 

ilCounply 

2 

Low 

4500 

27400 

45 

367 

#12 

5  - 

NoriMntriFilt 

4 

High 

1500 

(2890^ 
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“Best-Value  Package” 

•Incremental/Cumulative 

costs 

•Line  12 
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Workgroup  Analysis 
Framework 


nEDGEWOOD 


•  Intended 

-  Repeat  process  used  in  BAM’s  initial  assessment 

-  Validate  BAM’s  results 

•  Actual 

-  Workgroup  restructured  the  decision  model 

-  Examined  a  new  set  of  funding  strategies  &  all  of 
their  possible  combinations 
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Workgroup  AnalysisI 
Framework 

Structure  of  Investment  Areas  &  Levels 


EOGEWOOD 


v  jj  EQUITY  for  Windows  -  [  c:\mydocu-1\other\equity\cpmast-1\cp_ap_03.Bqw] 


File  Edit  View  Confiipure  Clipboard  Analysis  Help 


1 

’ 

2 

3 

A 

5 

;  02Supply 

None 

Low 

Mod 

High 

|Mew3mum 

CatOx 

None 

Low 

Mod 

High 

iMaximum 

:  Regen 

None 

Low 

Mod 

High 

— .J^ 

|,Maximunn 

1  Single  Pass 

'  None  j 

Low 

Mod 

High 

iMawmgm 

Aero  Part  Rem 

None  1 

. . 1 

[  Low 

Mod 

High 

iMaximum 
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Workgroup  Results  ^  0s 


EOGEWOOD 


BFNffITS 


Ffortnf  P»fV!  EM 

TECH  LCVfL 


PT#f#riflc*V|tu»a 

COSTS  eCNEfiTS 

Cost 

Total  Total 


r,rn  r/,.r  pr* 

wi  .viir  I  :'4?  M? 


Nr<lPt-!.a.rr(iF; 

PfWiotr;  Pack  *3 


19&TI  iWn  F/.T  «■.? 

1»Q0  S6S  565 


Pareto  Diagram 


Frontier  Package 
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Workgroup  Results 


nEDGEWOOD 


Order  of  Buy 


c 


Order  of  Buy 


Order  of  Buy 


TECH  LEVEL 


JSC 

t  - 

Ci2  Supply 

1  None 

JSC 

2  , 

C.ilO.f 

1  None 

JSC 

3 

Per^pti 

1  None 

JSC 

4 

Sinqle 

1  None 

JSC 

5- 

Aero  Pott  Re;r’ 

1  None 

llllllllllllll^^ 

COSTS  BENEFITS 

INC  CUM  INC  CUM 

0  0  0  0 

0  0  0  0 

0  0  0  0 

0  0  0  0 

0  0  0  0 


m 

4  - 

Single  Po;-i 

2 

Low 

2500 

2500 

204 

204 

JSC 

3- 

Reuen 

2 

Low 

6500 

9000 

247 

452 

JSC 

5 

Aero  Part  Pern 

2 

Low 

3000 

12000 

103 

555 

JS«4 

4  • 

Single  Pose 

3 

Mod 

2500 

Cj45^ 

38 

Oigi" 

“Best-Value  Package” 
•Line  4 

•Workgroup  suggested  for 
BAM  to  follow  this 
investment  strategy 
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Lessons  Learned 


nEOGEWOOD 


What  did  eo  well? 

What  did  not  eo  well? 

•EQUITY^^  led  to  a  clear-cut 
solution  to  the  funding  strategy 
combinations 

•DPL^*^  did  not  lead  to  a  clear-cut 
decision  for  BAM  (limited  strategies) 

•The  BAM  was  provided  with  a 
rigorous  examination  of  the 

COLPRO  technology  areas 

•Working  groups  had  to  restructure 
the  BAM’s  models  (lack  of 
standardized  investment  categories) 

•An  assessment  of  the  funding 
strategies  was  accomplished  by  the 
BAM  and  the  Working  groups 

•Working  groups  could  not  validate 
BAM’s  decisions 
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